Quantization matrix compression in video coding

ABSTRACT

A method of quantization matrix compression in a video encoder is provided that includes preprocessing a quantization matrix by performing at least one selected from down-sampling the quantization matrix and imposing 135 degree symmetry on the quantization matrix, performing zigzag scanning on the pre-processed quantization matrix to generate a one dimensional (1D) sequence, predicting the 1D sequence to generate a residual 1D sequence, and coding the residual 1D sequence using kth order exp-Golomb coding to generate a compressed quantization matrix, wherein k≥0.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of application Ser. No. 13/305,560,filed Nov. 28, 2011, which claims benefit of U.S. Provisional PatentApplication Ser. No. 61/418,537, filed Dec. 1, 2010, U.S. ProvisionalPatent Application Ser. No. 61/494,312, filed Jun. 7, 2011, and U.S.Provisional Patent Application Ser. No. 61/550,797, filed Oct. 24, 2011,which are incorporated herein by reference in their entirety.

BACKGROUND OF THE INVENTION

Field of the Invention

Embodiments of the present invention generally relate to compression ofquantization matrices in video coding.

Description of the Related Art

Video compression, i.e., video coding, is an essential enabler fordigital video products as it enables the storage and transmission ofdigital video. In general, video compression techniques applyprediction, transformation, quantization, and entropy coding tosequential blocks of pixels in a video sequence to compress, i.e.,encode, the video sequence. Video decompression techniques generallyperform the inverse of these operations in reverse order to decompress,i.e., decode, a compressed video sequence.

In general, current video encoders break a picture into discrete blocks(e.g., 4×4 or 8×8) that are transformed into blocks of coefficients.Each block is then quantized by multiplying by a quantization scale anddividing element-wise by a quantization matrix. The video codingstandard supported by a video encoder defines default quantizationmatrices for the supported transform block sizes and may also allowcustom matrices to be used. If custom matrices are used, the matricesare compressed and transmitted in the coded bit stream for use indecoding.

Future video encoding standards such as the High Efficiency Video Coding(HEVC) standard currently under development by a Joint CollaborativeTeam on Video Coding (JCT-VC) established by the ISO/IEC Moving PictureExperts Group (MPEG) and ITU-T Video Coding Experts Group (VCEG) maysupport larger transform sizes (e.g., 16×16 and 32×32) in addition tothe smaller transform sizes of the current standards in order to improvecoding efficiency. Thus, quantization matrices of the same sizes willalso be needed. If quantization matrices vary from picture to picture,significant overhead to transmit the quantization matrices will beincurred if current compression techniques are used.

SUMMARY

Embodiments of the present invention relate to methods and apparatus forcompression of quantization matrices in video coding. In one aspect, amethod of compressing quantization matrices in a video encoder includespreprocessing a quantization matrix by performing at least one selectedfrom down-sampling the quantization matrix and imposing 135 degreesymmetry on the quantization matrix, performing zigzag scanning on thepre-processed quantization matrix to generate a one dimensional (1D)sequence, predicting the 1D sequence to generate a residual 1D sequence,and coding the residual 1D sequence using kth order exp-Golomb coding togenerate a compressed quantization matrix, wherein k≥0.

In one aspect, a method of decompressing a compressed quantizationmatrix in a video decoder includes decoding a coded residual onedimensional (1D) sequence using kth order exp-Golomb coding to generatea reconstructed 1D residual sequence, wherein k≥0, performing inverseprediction on the reconstructed 1D residual sequence to generate areconstructed 1D sequence, performing inverse zigzag scanning on thereconstructed 1D sequence to generate a reconstructed quantizationmatrix, and postprocessing the reconstructed quantization matrix byperforming at least one selected from up-sampling the reconstructedquantization matrix and applying 135 degree symmetry processing to thereconstructed quantization matrix.

In one aspect, a method of processing a compressed video bit stream in avideo decoder includes receiving a compressed quantization matrix,decompressing the compressed reference quantization matrix to generate afirst reconstructed quantization matrix, and up-sampling the firstreconstructed quantization matrix to generate a second reconstructedquantization matrix larger than the first reconstructed quantizationmatrix. The method may further include up-sampling the secondreconstructed quantization matrix to generate a third reconstructedquantization matrix larger than the second reconstructed quantizationmatrix.

BRIEF DESCRIPTION OF THE DRAWINGS

Particular embodiments will now be described, by way of example only,and with reference to the accompanying drawings:

FIG. 1 is a flow diagram illustrating the H.264 quantization matrixcompression method;

FIG. 2 is a block diagram of a digital system;

FIGS. 3A and 3B are block diagrams of a video encoder;

FIG. 4 is a block diagram of a video decoder;

FIG. 5 is a flow diagram of a method for quantization matrixcompression;

FIGS. 6-8C are examples;

FIGS. 9A and 9B are flow diagrams of methods for quantization matrixdecompression; and

FIG. 10 is a block diagram of an illustrative digital system.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

Specific embodiments of the invention will now be described in detailwith reference to the accompanying figures. Like elements in the variousfigures are denoted by like reference numerals for consistency.

For convenience of description, embodiments of the invention aredescribed herein in reference to the October 2010 draft specificationfor HEVC. The 2010 draft specification is entitled “WD1: Working Draft 1of High-Efficiency Video Coding” and is identified as document numberJCT-VC_C403. One of ordinary skill in the art will understand thatembodiments of the invention are not limited to this draft specificationor to HEVC.

Some aspects of this disclosure have been presented to the JCT-VC in thefollowing documents: JCTVC-D024. entitled “Compact Representation ofQuantization Matrices for HEVC”, Jan. 20-28, 2011, JCTVC-F085, entitled“Further Study on Compact Representation of Quantization Matrices”, Jul.14-22, 2011, and JCTVC-0083, entitled “CE4: Test Results on CompactRepresentation of Quantization Matrices”, Nov. 19-30, 2011. Thesedocuments are incorporated by reference herein in their entirety.

As used herein, the term “picture” refers to a frame or a field of aframe. A frame is a complete image captured during a known timeinterval. In HEVC, a largest coding unit (LCU) is the base unit used forblock-based coding. A picture is divided into non-overlapping LCUs. Thatis, an LCU plays a similar role in coding as the macroblock ofH.264/AVC, but it may be larger, e.g., 32×32, 64×64, etc. An LCU may bepartitioned into coding units (CU). A CU is a block of pixels within anLCU and the CUs within an LCU may be of different sizes. Thepartitioning is a recursive quadtree partitioning. The quadtree is splitaccording to various criteria until a leaf is reached, which is referredto as the coding node or coding unit. The maximum hierarchical depth ofthe quadtree is determined by the size of the smallest CU (SCU)permitted. The coding node is the root node of two trees, a predictiontree and a transform tree. A prediction tree specifies the position andsize of prediction units (PU) for a coding unit. A transform treespecifies the position and size of transform units (TU) for a codingunit. A transform unit may not be larger than a coding unit and the sizeof a transform unit may be 4×4, 8×8, 16×16, and 32×32. The sizes of thetransforms units and prediction units for a CU are determined by thevideo encoder during prediction based on minimization of rate/distortioncosts.

FIG. 1 illustrates the technique used to compress quantization matricestransmitted in the sequence parameter set (SPS) and picture parameterset (PPS) in the predecessor to HEVC, H.264/MPEG-4 AVC, referred to asH.264 herein. H.264 is a video coding standard developed by theInternational Telecommunication Union (ITU) TelecommunicationStandardization Sector (ITU-T) Video Coding Experts Group (VCEG)together with the ISO/IEC Moving Picture Experts Group (MPEG). The ITU-TH.264 standard and the ISO/IEC MPEG-4 AVC standard (formally, ISO/IEC14496-10-MPEG-4 Part 10, Advanced Video Coding) are jointly maintainedso that they have identical technical content.

As shown in FIG. 1, the compression technique takes a two-dimensional(2D) quantization matrix as input. Initially, zigzag scanning isperformed to convert the 2D matrix into a one-dimensional (1D) sequence.Then, 1D sample prediction is performed to create a 1D residualsequence. Finally, signed 0^(th) order exp-Golomb coding is performed onthe 1D residual sequence to generate the compressed output. Note thatthe compression is lossless.

H.264 supports 4×4 and 8×8 transform block sizes. Six 4×4 quantizationmatrices are defined for the 4×4 block size, i.e., 3 matrices forinter-coded Y, Cb, Cr components and three matrices for intra-coded Y,Cb, Cr components. Two 8×8 quantization matrices are defined for the 8×8block size, i.e., a matrix for intra-coded Y components and a matrix forinter-coded Y components. As currently defined, HEVC supports fourtransform block sizes, 4×4, 8×8, 16×16, and 32×32, and six quantizationmatrices for each block size, i.e., for each block size, there are 3matrices for inter-coded Y, Cb, Cr components and three matrices forintra-coded Y, Cb, Cr components. If the H.264 technique for compressingquantization matrices is used for HEVC, significant overhead will beincurred to include these matrices in a picture parameter set (PPS) or asequence parameter set (SPS).

In general, an SPS is a set of parameters to be applied to a videosequence and a PPS is a set of parameters to be applied to one or moreconsecutive pictures in a video sequence. In both H.264 and HEVC, ifquantization matrices other than those default matrices defined by thestandard are to be used at the sequence level, the matrices arecompressed and inserted the coded bit stream at the SPS level.Similarly, if quantization matrices other than those default matricesdefined by the standard or at the sequence level are to be used for oneor more consecutive pictures, the matrices are compressed and insertedin the coded bit stream at the PPS level.

Table 1 is a comparison of the overhead of including the H.264compressed quantization matrices in a parameter set to the overhead ofincluding the HEVC compressed quantization matrices in a parameter setassuming the H.264 quantization matrix compression technique of FIG. 1.The quantization matrices of Tables 2-5 are used for this comparison.Table 2 is an example 4×4 quantization matrix, Table 3 is an example 8×8quantization matrix, Table 4 is an example 16×16 quantization matrix,and Table 5 is an example 32×32 quantization matrix. The second columnof Table 1 shows the number of bits in the respective compressedquantization matrices, fourth column shows the total number of bitsassuming the 8 quantization matrices of H.264, and the sixth columnshows the total number of bits assuming the 24 quantization matrices ofHEVC. Note that total number of bits for HEVC is more than a 10×increase over that of H.264.

TABLE 1 Number of Total bits Total quantiza- for quanti- Number of bitsfor Quantiza- Num- tion matri- zation ma- quantization quantization tionmatrix ber of ces in trices in matrices in matrices in block size bitsH.264 H.264 HEVC HEVC 4 × 4  140 6 1528 6 16584 8 × 8  344 2 6 16 × 16 702 0 6 32 × 32 1578 0 6

TABLE 2  8,  20,  68,  80, 20,  56,  92, 152, 68,  92, 140, 164, 80,152, 164, 188,

TABLE 3 8, 11, 23, 26, 50, 53, 89, 92, 11, 20, 29, 47, 56, 86, 95, 134,23, 29, 44, 59, 83, 98, 131, 137, 26, 47, 59, 80, 101, 128, 140, 167,50, 56, 83, 101, 125, 143, 164, 170, 53, 86, 98, 128, 143, 161, 173,188, 89, 95, 131, 140, 164, 173, 185, 191, 92, 134, 137, 167, 170, 188,191, 197,

TABLE 4 8, 8, 11, 12, 18, 19, 28, 29, 41, 41, 56, 57, 75, 76, 97, 98, 8,11, 13, 17, 20, 27, 29, 40, 42, 56, 58, 74, 77, 96, 98, 120, 11, 13, 17,20, 26, 30, 39, 43, 55, 59, 74, 77, 95, 99, 119, 121, 12, 17, 20, 26,31, 38, 44, 54, 59, 73, 78, 95, 100, 119, 122, 140, 18, 20, 26, 31, 38,44, 53, 60, 72, 79, 94, 101, 118, 122, 140, 141, 19, 27, 30, 38, 44, 53,61, 71, 80, 93, 101, 117, 123, 139, 142, 158, 28, 29, 39, 44, 53, 61,71, 80, 92, 102, 116, 124, 138, 143, 157, 158, 29, 40, 43, 54, 60, 71,80, 92, 103, 116, 125, 137, 143, 156, 159, 172, 41, 42, 55, 59, 72, 80,92, 103, 115, 125, 137, 144, 155, 160, 171, 173, 41, 56, 59, 73, 79, 93,102, 116, 125, 136, 145, 155, 161, 170, 173, 183, 56, 58, 74, 78, 94,101, 116, 125, 137, 145, 154, 161, 170, 174, 182, 184, 57, 74, 77, 95,101, 117, 124, 137, 144, 155, 161, 169, 175, 182, 185, 191, 75, 77, 95,100, 118, 123, 138, 143, 155, 161, 170, 175, 181, 185, 191, 192, 76, 96,99, 119, 122, 139, 143, 156, 160, 170, 174, 182, 185, 190, 193, 197, 97,98, 119, 122, 140, 142, 157, 159, 171, 173, 182, 185, 191, 193, 196,197, 98, 120, 121, 140, 141, 158, 158, 172, 173, 183, 184, 191, 192,197, 197, 199,

TABLE 5 8, 8, 8, 9, 10, 10, 13, 13, 16, 16, 20, 20, 24, 25, 30, 30, 36,36, 43, 43, 51, 51, 59, 59, 68, 68, 78, 78, 89, 89, 100, 101, 8, 8, 9,10, 11, 12, 13, 16, 16, 20, 20, 24, 25, 30, 30, 36, 36, 43, 43, 50, 51,59, 59, 68, 69, 78, 79, 89, 89, 100, 101, 112, 8, 9, 10, 11, 12, 13, 15,16, 19, 20, 24, 25, 29, 30, 36, 37, 43, 44, 50, 51, 59, 60, 68, 69, 78,79, 89, 89, 100, 101, 112, 112, 9, 10, 11, 12, 13, 15, 17, 19, 20, 24,25, 29, 31, 35, 37, 42, 44, 50, 51, 59, 60, 68, 69, 78, 79, 88, 90, 100,101, 112, 113, 123, 10, 11, 12, 13, 15, 17, 19, 21, 24, 25, 29, 31, 35,37, 42, 44, 50, 52, 58, 60, 68, 69, 77, 79, 88, 90, 100, 101, 112, 113,123, 123, 10, 12, 13, 15, 17, 19, 21, 23, 26, 29, 31, 35, 37, 42, 44,50, 52, 58, 60, 67, 69, 77, 79, 88, 90, 99, 101, 111, 113, 123, 124,134, 13, 13, 15, 17, 19, 21, 23, 26, 29, 31, 35, 37, 42, 44, 50, 52, 58,60, 67, 70, 77, 80, 88, 90, 99, 102, 111, 113, 123, 124, 133, 134, 13,16, 16, 19, 21, 23, 26, 29, 31, 35, 38, 42, 44, 49, 52, 58, 61, 67, 70,77, 80, 88, 90, 99, 102, 111, 113, 122, 124, 133, 134, 143, 16, 16, 19,20, 24, 26, 29, 31, 35, 38, 41, 45, 49, 52, 58, 61, 67, 70, 77, 80, 87,91, 99, 102, 111, 113, 122, 124, 133, 134, 143, 143, 16, 20, 20, 24, 25,29, 31, 35, 38, 41, 45, 49, 53, 57, 61, 67, 70, 77, 80, 87, 91, 99, 102,111, 114, 122, 124, 133, 134, 143, 143, 152, 20, 20, 24, 25, 29, 31, 35,38, 41, 45, 49, 53, 57, 61, 66, 70, 76, 80, 87, 91, 98, 102, 110, 114,122, 125, 133, 134, 143, 144, 152, 152, 20, 24, 25, 29, 31, 35, 37, 42,45, 49, 53, 57, 61, 66, 71, 76, 80, 87, 91, 98, 103, 110, 114, 122, 125,132, 135, 142, 144, 152, 152, 160, 24, 25, 29, 31, 35, 37, 42, 44, 49,53, 57, 61, 66, 71, 76, 81, 87, 91, 98, 103, 110, 114, 122, 125, 132,135, 142, 144, 151, 152, 160, 160, 25, 30, 30, 35, 37, 42, 44, 49, 52,57, 61, 66, 71, 76, 81, 86, 92, 98, 103, 110, 114, 121, 125, 132, 135,142, 144, 151, 153, 160, 160, 167, 30, 30, 36, 37, 42, 44, 50, 52, 58,61, 66, 71, 76, 81, 86, 92, 98, 103, 110, 115, 121, 125, 132, 135, 142,144, 151, 153, 159, 161, 167, 167, 30, 36, 37, 42, 44, 50, 52, 58, 61,67, 70, 76, 81, 86, 92, 98, 103, 110, 115, 121, 125, 132, 135, 142, 145,151, 153, 159, 161, 167, 168, 174, 36, 36, 43, 44, 50, 52, 58, 61, 67,70, 76, 80, 87, 92, 98, 103, 109, 115, 121, 126, 131, 136, 141, 145,151, 153, 159, 161, 167, 168, 174, 174, 36, 43, 44, 50, 52, 58, 60, 67,70, 77, 80, 87, 91, 98, 103, 110, 115, 121, 126, 131, 136, 141, 145,150, 153, 159, 161, 167, 168, 173, 174, 180, 43, 43, 50, 51, 58, 60, 67,70, 77, 80, 87, 91, 98, 103, 110, 115, 121, 126, 131, 136, 141, 145,150, 154, 159, 161, 166, 168, 173, 174, 179, 180, 43, 50, 51, 59, 60,67, 70, 77, 80, 87, 91, 98, 103, 110, 115, 121, 126, 131, 136, 141, 145,150, 154, 158, 161, 166, 168, 173, 175, 179, 180, 185, 51, 51, 59, 60,68, 69, 77, 80, 87, 91, 98, 103, 110, 114, 121, 125, 131, 136, 141, 145,150, 154, 158, 162, 166, 169, 173, 175, 179, 180, 185, 185, 51, 59, 60,68, 69, 77, 80, 88, 91, 99, 102, 110, 114, 121, 125, 132, 136, 141, 145,150, 154, 158, 162, 166, 169, 173, 175, 179, 180, 184, 185, 189, 59, 59,68, 69, 77, 79, 88, 90, 99, 102, 110, 114, 122, 125, 132, 135, 141, 145,150, 154, 158, 162, 166, 169, 173, 175, 179, 181, 184, 185, 189, 189,59, 68, 69, 78, 79, 88, 90, 99, 102, 111, 114, 122, 125, 132, 135, 142,145, 150, 154, 158, 162, 166, 169, 172, 175, 179, 181, 184, 185, 189,189, 193, 68, 69, 78, 79, 88, 90, 99, 102, 111, 114, 122, 125, 132, 135,142, 145, 151, 153, 159, 161, 166, 169, 173, 175, 178, 181, 184, 186,188, 190, 192, 193, 68, 78, 79, 88, 90, 99, 102, 111, 113, 122, 125,132, 135, 142, 144, 151, 153, 159, 161, 166, 169, 173, 175, 179, 181,184, 186, 188, 190, 192, 193, 195, 78, 79, 89, 90, 100, 101, 111, 113,122, 124, 133, 135, 142, 144, 151, 153, 159, 161, 166, 168, 173, 175,179, 181, 184, 186, 188, 190, 192, 193, 195, 196, 78, 89, 89, 100, 101,111, 113, 122, 124, 133, 134, 142, 144, 151, 153, 159, 161, 167, 168,173, 175, 179, 181, 184, 186, 188, 190, 192, 193, 195, 196, 197, 89, 89,100, 101, 112, 113, 123, 124, 133, 134, 143, 144, 151, 153, 159, 161,167, 168, 173, 175, 179, 180, 184, 185, 188, 190, 192, 193, 195, 196,197, 198, 89, 100, 101, 112, 113, 123, 124, 133, 134, 143, 144, 152,152, 160, 161, 167, 168, 173, 174, 179, 180, 184, 185, 189, 190, 192,193, 195, 196, 197, 198, 199, 100, 101, 112, 113, 123, 124, 133, 134,143, 143, 152, 152, 160, 160, 167, 168, 174, 174, 179, 180, 185, 185,189, 189, 192, 193, 195, 196, 197, 198, 199, 199, 101, 112, 112, 123,123, 134, 134, 143, 143, 152, 152, 160, 160, 167, 167, 174, 174, 180,180, 185, 185, 189, 189, 193, 193, 195, 196, 197, 198, 199, 199, 199,

Embodiments of the invention provide for compression of quantizationmatrices to reduce the number of bits needed to represent the compressedmatrices. Further, unlike the H.264 technique, lossy compression may beperformed. More specifically, in some embodiments, zigzag scanning isperformed on a quantization matrix that may be optionally pre-processedby down-sampling, 135 degree symmetry processing, and/or 45 degreesymmetry processing. After the zigzag scanning, in some embodiments. oneof 1D prediction for signed exp-Golomb coding, 1D prediction forunsigned exp-Golomb coding, or matrix prediction from a reference matrixmay be selectively performed, followed by signed or unsigned kth orderexp-Golomb coding as appropriate to generate the final compressedquantization matrix. In some embodiments, rather than compressing largeblock size quantization matrices, e.g., 16×16 and 32×32 quantizationmatrices, and encoding them in the compressed video bit stream, suchmatrices may be derived from up-sampling smaller quantization matrices.

Further, the matrix compression may be performed in a manner that isbackward compatible with H.264. That is, for 4×4 and 8×8 matrices, ifnone of the preprocessing is performed, the zigzag scanning is the sameas that of the H.264 compression technique and the option of 1Dprediction for signed exp-Golomb coding following by signed 0^(th) orderexp-Golomb coding may be selected. However, example embodiments of thequantization matrix compression described herein differ from that of theH.264 matrix compression in several respects. For example, the use ofsigned or unsigned kth order exp-Golomb code for encoding predictionresiduals is permitted rather than being restricted to the signed 0^(th)order exp-Golomb code which improves compression efficiency. Further,the options of reducing the compressed matrix size by imposing symmetryalong 135 degrees and/or symmetry (with offset) along 45 degrees areprovided. The option of down-sampling quantization matrices to furtherdecrease the compression ratio is also provided. Finally, the option ofusing matrix prediction from other quantization matrices, either thedefault matrices or matrices that have been previously compressed andtransmitted is provided.

Compared to the H.264 quantization matrix compression technique, someembodiments of the quantization matrix compression described herein mayprovide approximately 7× better compression efficiency with the use ofmatrix down-sampling, the imposition of symmetry on the quantizationmatrix along 135 and 45 (with offset) degrees, and the use of theunsigned kth order exp-Golomb code instead of the signed 0^(th) orderexp-Golomb code. Embodiments provide flexibility for an encoder tochoose the desired compression level.

FIG. 2 shows a block diagram of a digital system that includes a sourcedigital system 200 that transmits encoded video sequences to adestination digital system 202 via a communication channel 216. Thesource digital system 200 includes a video capture component 204, avideo encoder component 206, and a transmitter component 208. The videocapture component 204 is configured to provide a video sequence to beencoded by the video encoder component 206. The video capture component204 may be, for example, a video camera, a video archive, or a videofeed from a video content provider. In some embodiments, the videocapture component 204 may generate computer graphics as the videosequence, or a combination of live video, archived video, and/orcomputer-generated video.

The video encoder component 206 receives a video sequence from the videocapture component 204 and encodes it for transmission by the transmittercomponent 208. The video encoder component 206 receives the videosequence from the video capture component 204 as a sequence of frames,divides the frames into largest coding units (LCUs), and encodes thevideo data in the LCUs. The video encoder component 206 may beconfigured to apply quantization matrix compression and decompressiontechniques during the encoding process as described herein. Anembodiment of the video encoder component 206 is described in moredetail herein in reference to FIGS. 3A and 3B.

The transmitter component 208 transmits the encoded video data to thedestination digital system 202 via the communication channel 216. Thecommunication channel 216 may be any communication medium, orcombination of communication media suitable for transmission of theencoded video sequence, such as, for example, wired or wirelesscommunication media, a local area network, or a wide area network.

The destination digital system 202 includes a receiver component 210, avideo decoder component 212 and a display component 214. The receivercomponent 210 receives the encoded video data from the source digitalsystem 200 via the communication channel 216 and provides the encodedvideo data to the video decoder component 212 for decoding. The videodecoder component 212 reverses the encoding process performed by thevideo encoder component 206 to reconstruct the LCUs of the videosequence. The video decoder component 212 may be configured to applyquantization matrix decompression techniques during the decoding processas described herein. An embodiment of the video decoder component 212 isdescribed in more detail below in reference to FIG. 4.

The reconstructed video sequence is displayed on the display component214. The display component 214 may be any suitable display device suchas, for example, a plasma display, a liquid crystal display (LCD), alight emitting diode (LED) display, etc.

In some embodiments, the source digital system 200 may also include areceiver component and a video decoder component and/or the destinationdigital system 202 may include a transmitter component and a videoencoder component for transmission of video sequences both directionsfor video steaming, video broadcasting, and video telephony. Further,the video encoder component 206 and the video decoder component 212 mayperform encoding and decoding in accordance with one or more videocompression standards. The video encoder component 206 and the videodecoder component 212 may be implemented in any suitable combination ofsoftware, firmware, and hardware, such as, for example, one or moredigital signal processors (DSPs), microprocessors, discrete logic,application specific integrated circuits (ASICs), field-programmablegate arrays (FPGAs), etc.

FIGS. 3A and 3B show block diagrams of an example video encoder. FIG. 3Ashows a high level block diagram of the video encoder and FIG. 3B showsa block diagram of the LCU processing component 342 of the videoencoder. As shown in FIG. 3A, a video encoder includes a coding controlcomponent 340, an LCU processing component 342, a rate control component344, a quantization matrix processing component 348, and a memory 346.The memory 346 may be internal memory, external memory, or a combinationthereof.

An input digital video sequence is provided to the coding controlcomponent 340. The coding control component 340 sequences the variousoperations of the video encoder. i.e., the coding control component 340runs the main control loop for video encoding. For example, the codingcontrol component 340 performs any processing on the input videosequence that is to be done at the frame level, such as determining thecoding type (I, P, or B) of a picture based on the high level codingstructure, e.g., IPPP, IBBP, hierarchical-B, and dividing a frame intoLCUs for further processing. The coding control component 340 also maydetermine the initial LCU CU structure for each CU and providesinformation regarding this initial LCU CU structure to the variouscomponents of the LCU processing component 342 as needed. The codingcontrol component 340 also may determine the initial PU and TU structurefor each CU and provides information regarding this initial structure tothe various components of the LCU processing component 342 as needed.

The rate control component 344 determines a quantization scale QS foreach CU in a picture based on various rate control criteria and providesthe QS to the coding control component 340 for communication to variouscomponents of the LCU processing component 342 as needed. The ratecontrol component 344 may use any suitable rate control algorithm.

The quantization matrix processing component 348 provides functionalityfor compression and decompression of quantization matrices. Compressionand decompression of quantization matrices is described below inreference to FIGS. 5-9.

The coding control component 340 determines the quantization matrix tobe used for quantizing a transform unit. As was previously mentioned, inHEVC, there are four TU block sizes and 6 default quantization matricesfor each TU block size. Further, any or all of these default matricesmay be replaced by custom quantization matrices, i.e., non-defaultquantization matrices at the sequence level and/or at the picture level.At the beginning of a video sequence, the coding control component 340may determine whether the default quantization matrices are to be usedat the sequence level or whether one or more of the sequence levelquantization matrices are to be non-default quantization matrices. Ifone or more non-default quantization matrices are to be used at thesequence level, the coding control component 340 may cause thequantization matrix processing component 348 to compress the one or morematrices for inclusion in the SPS.

Further, the coding control component 340 may also determine whether thesequence level quantization matrices are to be used at the picture levelor whether one or more of the picture level quantization matrices are tobe different. If one or more quantization matrices different from thesequence level quantization matrices are to be used in encoding of apicture or sequence of pictures, the coding control component 340 maycause the quantization matrix processing component 348 to compress theone or more matrices for inclusion in the PPS for the picture orsequence of pictures.

In some embodiments, rather than compressing a large block sizequantization matrix for inclusion in the SPS or a PPS, a convention maybe used in which if a smaller, e.g., 8×8, non-default quantizationmatrix is compressed and included in the SPS or a PPS, the decoder willautomatically derive larger, e.g., 16×16 and/or 32×32, quantizationmatrices of the same type (inter/intra Y, Cr, Cb) from the smallernon-default quantization matrix by up-sampling the smaller quantizationmatrix. Any suitable technique for selecting the reference quantizationmatrix may be used.

As is explained in more detail below, if a non-default quantizationmatrix is to be used for quantization of a TU, a reconstructed versionof the matrix may be used rather than the original matrix. This isbecause the matrix compression process may be lossy. If the encoder usesthe original matrix and the decoder uses the reconstructed matrix, therewill be a mismatch. Accordingly, if a non-default quantization matrix isto be used for a TU, the coding control component 340 may request areconstructed quantization matrix from the quantization matrixprocessing component 348. In some embodiments, the reconstruction may beperformed if the compression of the quantization matrix may be lossy andnot performed if the compression is lossless.

In some embodiments, the quantization matrix processing component 348may compress and decompress the non-default quantization matrix togenerate the reconstructed quantization matrix, and provides thereconstructed quantization matrix to the coding control component 340for communication to the quantize component 306 and the dequantizecomponent 312 (see FIG. 3B). In some embodiments, the quantizationmatrix processing component 348 may store reconstructed quantizationmatrices in the memory 346 to avoid recreating the matrices each timethey are requested. In some embodiments, when the previously mentionedconvention to avoid inclusion of large block size quantization matricesin the SPS or a PPS is used, the quantization matrix processingcomponent 340 may up-sample the smaller non-default quantization matrixto generate the reconstructed quantization matrix. An example ofup-sampling of quantization matrices is described below in reference toFIGS. 8A-8C.

The LCU processing component 342 receives LCUs of the input videosequence from the coding control component 340 and encodes the LCUsunder the control of the coding control component 340 to generate thecompressed video stream. The CUs in the CU structure of an LCU may beprocessed by the LCU processing component 342 in a depth-first Z-scanorder.

FIG. 3B shows the basic coding architecture of the LCU processingcomponent 342. The LCUs 300 from the coding control unit 340 areprovided as one input of a motion estimation component 320, as one inputof an intra prediction component 324, and to a positive input of acombiner 302 (e.g.. adder or subtractor or the like). Further, althoughnot specifically shown, the prediction mode of each picture as selectedby the coding control component 340 is provided to a mode selectorcomponent, and the entropy encoder 334.

The storage component 318 provides reference data to the motionestimation component 320 and to the motion compensation component 322.The reference data may include one or more previously encoded anddecoded CUs, i.e., reconstructed CUs.

The motion estimation component 320 provides motion estimationinformation to the motion compensation component 322 and the entropyencoder 334. More specifically, the motion estimation component 320performs tests on CUs in an LCU based on multiple temporal predictionmodes and transform block sizes using reference data from storage 318 tochoose the best motion vector(s)/prediction mode based on a coding cost.To perform the tests, the motion estimation component 320 may begin withthe CU structure provided by the coding control component 340. Themotion estimation component 320 may divide each CU indicated in the CUstructure into prediction units according to the unit sizes ofprediction modes and into transform units according to the transformblock sizes and calculate the coding costs for each prediction mode andtransform block size for each CU.

For coding efficiency, the motion estimation component 320 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best motionvectors/prediction modes, in addition to testing with the initial CUstructure, the motion estimation component 320 may also choose to dividethe larger CUs in the initial CU structure into smaller CUs (within thelimits of the recursive quadtree structure), and calculate coding costsat lower levels in the coding hierarchy. If the motion estimationcomponent 320 changes the initial CU structure, the modified CUstructure is communicated to other components in the LCU processingcomponent 342 that need the information.

The motion estimation component 320 provides the selected motion vector(MV) or vectors and the selected prediction mode for each interpredicted CU to the motion compensation component 323 and the selectedmotion vector (MV) to the entropy encoder 334. The motion compensationcomponent 322 provides motion compensated inter prediction informationto the mode decision component 326 that includes motion compensatedinter predicted CUs, the selected temporal prediction modes for theinter predicted CUs, and corresponding transform block sizes. The codingcosts of the inter predicted CUs are also provided to the mode decisioncomponent 326.

The intra prediction component 324 provides intra prediction informationto the mode decision component 326 that includes intra predicted CUs andthe corresponding spatial prediction modes. That is, the intraprediction component 324 performs spatial prediction in which testsbased on multiple spatial prediction modes and transform unit sizes areperformed on CUs in an LCU using previously encoded neighboring CUs ofthe picture from the buffer 328 to choose the best spatial predictionmode for generating an intra predicted CU based on a coding cost. Toperform the tests, the intra prediction component 324 may begin with theCU structure provided by the coding control component 340. The ultraprediction component 324 may divide each CU indicated in the CUstructure into prediction units according to the unit sizes of thespatial prediction modes and into transform units according to thetransform block sizes and calculate the coding costs for each predictionmode and transform block size for each CU.

For coding efficiency, the intra prediction component 324 may alsodecide to alter the CU structure by further partitioning one or more ofthe CUs in the CU structure. That is, when choosing the best predictionmodes, in addition to testing with the initial CU structure, the intraprediction component 324 may also chose to divide the larger CUs in theinitial CU structure into smaller CUs (within the limits of therecursive quadtree structure), and calculate coding costs at lowerlevels in the coding hierarchy. If the intra prediction component 324changes the initial CU structure, the modified CU structure iscommunicated to other components in the LCU processing component 342that need the information. Further, the coding costs of the intrapredicted CUs and the associated transform block sizes are also providedto the mode decision component 326.

The mode decision component 326 selects between the motion-compensatedinter predicted CUs from the motion compensation component 322 and theintra predicted CUs from the intra prediction component 324 based on thecoding costs of the CUs and the picture prediction mode provided by themode selector component. The output of the mode decision component 326,i.e., the predicted CU, is provided to a negative input of the combiner302 and to a delay component 330. The associated transform block size isalso provided to the transform component 304. The output of the delaycomponent 330 is provided to another combiner (i.e., an adder) 338. Thecombiner 302 subtracts the predicted CU from the current CU to provide aresidual CU to the transform component 304. The resulting residual CU isa set of pixel difference values that quantify differences between pixelvalues of the original CU and the predicted CU.

The transform component 304 performs block transforms on the residualCUs to convert the residual pixel values to transform coefficients andprovides the transform coefficients to a quantize component 306. Thetransform component 304 receives the transform block sizes for theresidual CUs and applies transforms of the specified sizes to the CUs togenerate transform coefficients.

The quantize component 306 quantizes the transform coefficients based onquantization parameters (QPs) and the quantization matrices provided bythe coding control component 340 and the transform sizes. The quantizedtransform coefficients are taken out of their scan ordering by a scancomponent 308 and arranged by significance, such as, for example,beginning with the more significant coefficients followed by the lesssignificant.

The ordered quantized transform coefficients for a CU provided via thescan component 308 along with header information for the CU are coded bythe entropy encoder 334, which provides a compressed bit stream to avideo buffer 336 for transmission or storage. The header information mayinclude an indicator of the transform block size used for the CU and thequantization parameter for the CU. The entropy encoder 334 also codesthe CU structure of each LCU.

Inside the LCU processing component 342 is an embedded decoder. As anycompliant decoder is expected to reconstruct an image from a compressedbit stream, the embedded decoder provides the same utility to the videoencoder. Knowledge of the reconstructed input allows the video encoderto transmit the appropriate residual energy to compose subsequentframes. To determine the reconstructed input, i.e., reference data, theordered quantized transform coefficients for a CU provided via the scancomponent 308 are returned to their original post-transform arrangementby an inverse scan component 310, the output of which is provided to adequantize component 312, which outputs a reconstructed version of thetransform result from the transform component 304.

The dequantized transform coefficients are provided to the inversetransform component 314, which outputs estimated residual informationwhich represents a reconstructed version of a residual CU. The inversetransform component 314 receives the transform block size used togenerate the transform coefficients and applies inverse transform(s) ofthe specified size to the transform coefficients to reconstruct theresidual values.

The reconstructed residual CU is provided to the combiner 338. Thecombiner 338 adds the delayed selected CU to the reconstructed residualCU to generate an unfiltered reconstructed CU, which becomes part ofreconstructed picture information. The reconstructed picture informationis provided via a buffer 328 to the intra prediction component 324 andto a filter component 316. The filter component 316 is an in-loop filterwhich filters the reconstructed frame information and provides filteredreconstructed CUs, i.e., reference data, to the storage component 318.

FIG. 4 shows a block diagram of an example video decoder. The videodecoder operates to reverse the encoding operations, i.e., entropycoding, quantization, transformation, and prediction, performed by thevideo encoder of FIGS. 3A and 3B to regenerate the frames of theoriginal video sequence. In view of the above description of a videoencoder, one of ordinary skill in the art will understand thefunctionality of components of the video decoder without detailedexplanation.

The entropy decoding component 400 receives an entropy encoded(compressed) video bit stream and reverses the entropy coding to recoverthe encoded CUs and header information such as the quantizationparameters, the transform block sizes, compressed quantization matrices(if any), and the encoded CU structures of the LCUs. The quantizationmatrix decompression component 401 decompresses any compressedquantization matrices to created reconstructed quantization matrices andprovides the reconstructed quantization matrices to the inversequantization component 402. Decompression of compressed quantizationmatrices is described below in reference to FIGS. 9A and 9B. In someembodiments, a compressed quantization matrix may not be physicallypresent in the compressed bit stream for some large block sizequantization matrices. Instead, the quantization matrix decompressioncomponent 401 may reconstruct a large block size quantization matrix byup-sampling a smaller non-default quantization matrix. An example ofup-sampling of quantization matrices is described below in reference toFIGS. 8A-8C.

The inverse quantization component 402 de-quantizes the quantizedtransform coefficients of the residual CUs. The inverse transformcomponent 404 transforms the frequency domain data from the inversequantization component 402 back to residual CUs. That is, the inversetransform component 404 applies an inverse unit transform, i.e., theinverse of the unit transform used for encoding, to the de-quantizedresidual coefficients to produce the residual CUs.

A residual CU supplies one input of the addition component 406. Theother input of the addition component 406 comes from the mode switch408. When inter-prediction mode is signaled in the encoded video stream,the mode switch 408 selects a prediction unit from the motioncompensation component 410 and when intra-prediction is signaled, themode switch selects a prediction unit from the intra predictioncomponent 414. The motion compensation component 410 receives referencedata from storage 412 and applies the motion compensation computed bythe encoder and transmitted in the encoded video bit stream to thereference data to generate a predicted CU. The intra-predictioncomponent 414 receives previously decoded predicted CUs from the currentpicture and applies the intra-prediction computed by the encoder assignaled by a spatial prediction mode transmitted in the encoded videobit stream to the previously decoded predicted CUs to generate apredicted CU.

The addition component 406 generates a decoded CU, by adding theselected predicted CU and the residual CU. The output of the additioncomponent 406 supplies the input of the in-loop filter component 416.The in-loop filter component 416 smoothes artifacts created by the blocknature of the encoding process to improve the visual quality of thedecoded frame. The output of the in-loop filter component 416 is thedecoded frames of the video bit stream. Each decoded CU is stored instorage 412 to be used as reference data.

FIG. 5 shows a flow diagram of a method for compressing a quantizationmatrix in a video encoder. An embodiment of the method may be performed,for example, by the quantization matrix processing component 348 of FIG.3. The input to the method is a 2D N×N quantization matrix and theoutput is a compressed quantization matrix. The description of themethod includes example pseudo code illustrating the functionality ofvarious aspects of the method. In this pseudo code. {Qmatx(i, j), fori=0, 1, 2, . . . , N−1, j=0, 1, 2, . . . , N−1} is the N×N quantizationmatrix.

The method includes three optional preprocessing operations that may beperformed on the quantization matrix, down-sampling 500, 135 degreesymmetry processing 502, and 45 degree symmetry processing 504. A videoencoder may choose to perform none of these operations, any one of theseoperations, any two of these operations, or all three operations. Anysuitable criteria may be used for determining which, if any, of theseoperations is to be performed.

The down-sampling 500, if performed. down-samples the quantizationmatrix to reduce the resolution. For example, the N×N quantizationmatrix may be reduced to (N/2+1)×(N/2+1). Table 6 is pseudo codeillustrating the example down-sampling. In this pseudo code,{deciQmatx(i, j), for i=0, 1, 2, . . . , N/2, j=0, 1, 2, . . . , N/2} isthe down-sampled quantization matrix. Note that this down-sampling isnon-normative.

TABLE 6 for (i=0; i< N/2 +1; i++) { if (i > 1) row = i*2 −1; else row =i; for (j=0; j< N/2 +1; j++) { if (j > 1) col = 2*j −1; else col = j;deciQMatx(i, j) = Qmatx(row,col); }}

The 135 degree symmetry processing 502, if performed, so imposes matrixsymmetry along 135 degrees, i.e., {Qmatx(i, j)=Qmatx(j, i), for i=0, 1,2, . . . , N−1, j=0, 1, 2, . . . , N−1} if down-sampling 500 is notperformed or {deciQMatx (i, j)=deciQMatx (j, i), for i=0, 1, 2, . . . ,(N/2+1)−1, j=0, 1, 2, . . . , (N/2+1)−1} if down-sampling 500 isperformed.

The 45 degree symmetry processing 504, if performed, imposes matrixsymmetry with an offset along 45 degrees, i.e., {Qmatx(i,j)+Qmatx(N−1−j, N−1−i)=Qsum, for i=0, 1, 2, . . . , N−1, j=0, 1, 2, . .. , N−1} if down-sampling 500 is not performed or {deciQMatx (i,j)+deciQMatx ((N/2+1)−1−j, (N/2+1)−1−i)=Qsum, for i=0, 1, 2, . . . ,(N/2+1)−1, j=0, 1, 2, . . . , (N/2+1)−1} if down-sampling 500 isperformed. Qsum is the offset and is constant for the matrix.

The preprocessing performed to compress a quantization matrix needs tobe communicated in the encoded bit stream so that a decoder canappropriately reconstruct the quantization matrix. For example, a threebit flag may be used to signal which, if any, of the preprocessingoperations were performed. Further, if the 45 degree symmetry processing504 is performed, the offset used should also be communicated. Thisoffset may be encoded using unsigned kth order exp-Golomb coding.

Zigzag scanning 506 is performed to convert a 2D quantization matrix toa 1D sequence. The quantization matrix at this point may be the originalinput matrix if none of the preprocessing operations are performed ormay be the matrix resulting from performing one or more of thepreprocessing operations. Table 7 is example pseudo code for zigzagscanning. In this pseudo code, {zigzagScan(i, j), for i=0, 1, 2, . . . ,N−1, j=0, 1, 2, . . . , N−1} is the N×N zigzag scanning matrix, and{Qmatx1D(i), i=0, 1, 2, N*N−1} is the quantization matrix after zigzagscanning, i e , the 1D sequence. Note that if either of the symmetryprocessing operations has been performed, this pseudo code sets thosematrix entries that do not need to be encoded due to the imposedsymmetry to zero.

TABLE 7 for (i= 0; i < N; i++) for (j=0; j < N; j++) {  if (135 degreesymmetry processing is enabled && j < i) ∥ (45 degree symmetryprocessing is enabled && i + j >=N) Qmatx1D(zigzagScan[i, j]) =0;  elseQmatx1D(zigzagScan[i, j]) = Qmatx(i, j);}

The pseudo code of Table 7 assumes that down-sampling was not performed.If the original N×N matrix was down-sampled, the size of the matrixinput to zigzag scanning 506 is (N/2+1)×(N/2+1). For zigzag scanning ofthe smaller matrix, N is replaced by N/2+1 in the pseudo code. Further,the zigzag scanning matrix zigzagScan(i, j) is of size (N/2+1)×(N/2+1)and the length of the 1D sequence Qmatx1D(i) is (N/2+1)×(N/2+1).

After the zigzag scanning 506, one of three options for predicting andcoding the 1D sequence is selected 508 based on a prediction type chosenby the video encoder. Any suitable criteria may be used for determiningwhich of the three options for predicting and coding is to be selected.

Signed 1D prediction 510 creates a 1D residual sequence from the 1Dsequence, i.e., Qmatx1D, for signed exp-Golomb coding. Table 8 is pseudocode illustrating the creation of the 1D residual sequence. In thispseudo code, {residual (i), i=0, 1, 2, . . . , count−1} is the resulting1D residual sequence. Note that any zero entries in the 1D sequence areignored. The resulting 1D residual sequence is then encoded using signedkth order exp-Golomb coding 512 to generate the compressed quantizationmatrix. The value of k may be determined in any suitable way. Note thatin the pseudo code. the variable count stores the number of non-zerovalues in the 1D residual sequence. This count is needed for theexp-Golomb coding.

TABLE 8 Pred = 0; idx = 0;  For (i = 0; i < N*N; i++)  If (Qmatx1D[i]!=0) { residual[idx] = Qmatx1D[i] − pred; pred += residual[idx]; idx++;} count = idx;

Unsigned 1D prediction 514 creates a 1D residual sequence from the 1Dsequence for unsigned exp-Golomb coding. Table 9 is pseudo codeillustrating the creation of the 1D residual sequence. In this pseudocode, {residual (i), i=0, 1, 2, . . . , count−1} is the resulting 1Dresidual sequence. In unsigned exp-Golomb coding, all the residuals arerequired to be 0 or positive. Therefore, the pseudo code of Table 9forces all negative residuals to 0. Note that any zero entries in the 1Dsequence are ignored. The resulting 1D residual sequence is then encodedusing unsigned kth order exp-Golomb coding 516 to generate thecompressed quantization matrix. The value of k may be determined in anysuitable way. Note that in the pseudo code, the variable count storesthe number of non-zero values in the 1D residual sequence. This count isneeded for the exp-Golomb coding.

TABLE 9  Pred = 0; idx = 0;  For (i = 0; i < N*N; i++)  If (Qmatx1D[i]!=0) { residual[idx] = Qmatx1D[i] − pred; if (residual[idx] <0)residual[idx] = 0; pred += residual[idx]; idx++; } count = idx;

Matrix prediction 518 creates a 1D residual sequence from the 1Dsequence using a reference quantization matrix. Table 10 is pseudo codeillustrating the creation of the 1D residual sequence. In this pseudocode, {residual (i), i=0, 1, 2, . . . , count−1} is the resulting 1Dresidual sequence and {refQmatx(i,j), i=0, 1, 2, . . . , N−1, j=0, 1, 2,. . . , N−1} is the reference quantization matrix. Note that any zeroentries in the 1D sequence are ignored. The resulting 1D residualsequence is then encoded 520 to generate the compressed quantizationmatrix. Unsigned kth order exp-Golomb coding is used if all theresiduals in the sequence are positive and signed kth order exp-Golombcoding is used otherwise. The value of k may be determined in anysuitable way.

TABLE 10 // convert the reference quantization matrix to zigzag orderfor (i= 0; i < N; i++) for (j=0; j < N; j++) { refQmatx1D(zigzagScan[i,j]) = refQmatx(i, j)} // generate residual sequence  For (i = 0; i <N*N; i++)  If (Qmatx1D[i] !=0) {  residual[idx] = Qmatx1D[i] −refQmatx1D[i];  idx++; } count = idx;

For the SPS, a reference quantization matrix may be selected from any ofthe quantization matrices already signaled in the SPS. For example,consider the example SPS format in the pseudo code of Table 11. Apossible format for encoding quantization matrices is shown in lines31-35. In the pseudo code, seq_scaling_matrix_present_flag=0 indicatesthat the default quantization matrices are used. Otherwise, non-defaultquantization matrices may be present in SPS. Further,seq_scaling_list_present_flag[i]=0 indicates that a default quantizationmatrix is used for given scaling list index. Otherwise, a correspondingnon-default matrix is included in the SPS. The index i is in the rangeof [0:23] inclusive. A scaling list is a quantization matrix. Table 12shows a mapping for purposes of this example of the HEVC scaling listsrelative to transform block sizes, scaling list indices, intra/intercoding mode, and Y/CB/CR components.

Referring again to Table 11, scaling_list ( ) is a function whichcarries the compressed quantization matrix, and ScalingList4×4,ScalingList8×8, ScalingList16×16 and ScalingList32×32 are non-defaultquantization matrices for transform block sizes 4×4, 8×8, 16×16 and32×32, respectively. In addition, qmatx_compressionID is a 3-bit flagindicating which of the three preprocessing options, if any, wasperformed during compression of the scaling list. In this example, themost significant bit signals matrix down-sampling, the middle bitsignals 45 degree symmetry processing, and the least significant bitsignals 135 degree symmetry processing. Finally. log2_min_transform_block_size_minus2 and log2_diff_max_min_transform_block_size specify the transform sizes used anddetermine which of 24 possible non-default quantization matrices arepresent in SPS.

In this example, if one or more non-default quantization matrices areused, then the SPS will include information for each quantizationmatrix, whether default or non-default, and the information will beencoded in scaling list index order (see Table 12). As this portion ofthe SPS is generated and a non-default quantization matrix is compressedfor inclusion in the SPS, a reference quantization matrix for matrixprediction may be selected from any of the quantization matrices(default or non-default) already encoded in the SPS, regardless of size.If the selected reference quantization matrix is smaller than thequantization matrix being compressed, the reference quantization matrixis scaled up to the size of the size of the quantization matrix. Anysuitable up-sampling technique may be used. For example, the N/2×N/2 toN×N up-sampling technique described below in reference to FIGS. 8A-8Cmay be used. Any suitable criteria may be used for selection of thereference quantization matrix.

An indication of which matrix was used as the reference quantizationmatrix is included in the SPS to inform the decoder. This indication maybe, for example, the scaling list index of the matrix or the differencebetween the scaling list index of the quantization matrix and thescaling list index of the reference quantization matrix. The indicatormay be encoded with unsigned exp-Golomb code.

FIG. 6 shows an example of matrix prediction during generation of anSPS. Each of the blocks in this example corresponds to one of the 24quantization matrices and the index of each matrix is in thecorresponding block. The shaded blocks indicate quantization matricesalready encoded in the SPS and the unshaded blocks indicate quantizationmatrices yet to be encoded. Thus, in this example, quantization matrices0-14 are already encoded and quantization matrix 15, which is assumed tobe a non-default matrix, is the matrix currently being compressed usingmatrix prediction. Any of quantization matrices 0-14 may be used as thereference quantization matrix for quantization matrix 15. In thisexample, quantization matrix 12 has been selected as the referencequantization matrix. Thus, the reference matrix indicator included inthe SPS may be 15−12=3.

TABLE 11 Desc.  1 seq_parameter_set_rbsp( ) {  2 profile_idc u(8)  3reserved_zero_8bits /* equal to 0 */ u(8)  4 level_idc u(8)  5seq_parameter_set_id ue(v)  6 pic_width_in_luma_samples u(16)  7pic_height_in_luma_samples u(16)  8 bit_depth_luma_minus8 ue(v)  9bit_depth_chroma_minus8 ue(v) 10 log2_max_frame_num_minus4 ue(v) 11pic_order_cnt_type ue(v) 12 if( pic_order_cnt_type = = 0 ) 13log2_max_pic_order_cnt_lsb_minus4 ue(v) 14 else if( pic_order_cnt_type == 1 ) { 15 delta_pic_order_always_zero_flag u(1) 16offset_for_non_ref_pic se(v) 17 num_ref_frames_in_pic_order_cnt_cycleue(v) 18 for( i = 0; i < num_ref_frames_in_pic_order_cnt_cycle; i++ ) 19offset_for_ref_frame[ i ] se(v) 20 } 21 max_num_ref_frames ue(v) 22gaps_in_frame_num_value_allowed_flag u(1) 23log2_min_coding_block_size_minus3 ue(v) 24log2_diff_max_min_coding_block_size ue(v) 25log2_min_transform_block_size_minus2 ue(v) 26log2_diff_max_min_transform_block_size ue(v) 27max_transform_hierarchy_depth_inter ue(v) 28max_transform_hierarchy_depth_intra ue(v) 29adaptive_loop_filter_enabled_flag u(1) 30 cu_qp_delta_enabled_flag u(1)31 seq_scaling_matrix_present_flag u(1) 32 if(seq_scaling_matrix_present_flag ) 33 for( k = 0; k <=log2_diff_max_min_transform_block_size*6; k++ ) { 34  i = k +log2_min_transform_block_size_minus2*6 35 If ((i%6) == 0)qmatx_compressionID[i/6] u(3) 36 seq_scaling_list_present_flag[ i ] u(1)37 if( seq_scaling_list_present_flag[ i ] ) 38 if( i < 6 ) 39scaling_list( ScalingList4x4[ i ], 4, i, 1, qmatx_compressionID[i/6]) 40else (i < 12) 41 scaling_list( ScalingList8x8[ i − 6 ], 8, i, 1,qmatx_compressionID[i/6]) 42 else if (i < 18) 43 scaling_list(ScalingList16x16[ i − 12 ], 16, i, 1, qmatx_compressionID[i/6] ) 44 else45 scaling_list( ScalingList32x32[ i − 18 ], 32, i, 1,qmatx_compressionID[i/6] ) 46 } 47 rbsp_trailing_bits( ) 48 }

TABLE 12 Trans- form block Intra Inter size Y Cb Cr Y Cb Cr 4 × 4ScalingList ScalingList ScalingList ScalingList ScalingList ScalingList(scaling 4 × 4[0] 4 × 4[1] 4 × 4[2] 4 × 4[3] 4 × 4[4] 4 × 4[5] list 0 12 3 4 6 index) 8 × 8 ScalingList ScalingList ScalingList ScalingListScalingList ScalingList (scaling 8 × 8[0] 8 × 8[1] 8 × 8[2] 8 × 8[3] 8 ×8[4] 8 × 8[5] list 6 7 8 9 10 11 index) 16 × 16 ScalingList ScalingListScalingList ScalingList ScalingList ScalingList (scaling 16 × 16[0] 16 ×16[1] 16 × 16[2] 16 × 16[3] 16 × 16[4] 16 × 16[5] list 12 13 14 15 16 17index) 32 × 32 ScalingList ScalingList ScalingList ScalingListScalingList ScalingList (scaling 32 × 32[0] 32 × 32[1] 32 × 32[2] 32 ×32[3] 32 × 32[4] 32 × 32[5] list 18 19 20 21 22 23 index)

Referring again to matrix prediction 518 of FIG. 5, for a PPS, areference quantization matrix may be selected from any of thequantization matrices signaled in the SPS and from any of thequantization matrices already signaled in the PPS. For example, considerthe example PPS format in the pseudo code of Table 13. A possible formatfor encoding quantization matrices is shown in lines 10-23. In thispseudo code, pic_scaling_matrix_present_flag=0 indicates that thequantization matrices specified in SPS are used. Otherwise, thequantization matrices of SPS may be overwritten by the ones encoded inthe PPS. Further, pic_scaling_list_present_flag[i]=0 indicates that acorresponding quantization matrix defined in the SPS is used. Otherwise,a corresponding non-default matrix is included in the PPS whichoverwrites the one of the SPS. The index i is in the range of [0:23]inclusive. Scaling_list( ) ScalingList4×4, ScalingList8×8,ScalingList16×16, ScalingList32×32, qmatx_compressionID, log2_min_transform_block_size_minus2, and log2_diff_max_min_transform_block_size are previously defined in referenceto the example SPS format of Table 11.

Similar to the SPS, in this example, if one or more non-defaultquantization matrices are used, then the PPS will include informationfor each quantization matrix, whether default or non-default, and theinformation will be encoded in scaling list index order (see Table 12).As this portion of the PPS is generated and a non-default quantizationmatrix is compressed for inclusion in the PPS, a reference quantizationmatrix for matrix prediction may be selected from any of thequantization matrices (default or non-default) encoded in the SPS or anyof the quantization matrices (default or non-default) already encoded inthe PPS, regardless of size. If the selected reference quantizationmatrix is smaller or larger than the quantization matrix beingcompressed, the reference quantization matrix is scaled up or down tothe size of the size of the quantization matrix. Any suitableup-sampling technique may be used. For example, the N/2×N/2 to N×Nup-sampling technique described below in reference to FIGS. 8A-8C may beused. Further, any suitable down-sampling technique may be used to scaledown a reference quantization matrix. For example, a simpledown-sampling technique is to construct the smaller matrix by pickingevery other row and column from the N×N input matrix. Any suitablecriteria may be used for selection of the reference quantization matrix.

An indication of which matrix was used as the reference quantizationmatrix and an indication of whether the matrix is from the SPS or PPSare included in the PPS to inform the decoder. This reference matrixindicator may be, for example, the scaling list index of the matrix orthe difference between the scaling list index of the quantization matrixand the scaling list index of the reference quantization matrix. Thereference matrix indicator may be encoded with signed exp-Golomb code ifthe reference quantization matrix is selected from the SPS or unsignedexp-Golomb code if the reference quantization matrix is selected fromthe PPS.

FIG. 7 shows an example of matrix prediction during generation of a PPS.Both an SPS and a PPS are depicted. Each of the blocks in this examplecorresponds to one of the 24 quantization matrices and the index of eachmatrix is in the corresponding block. The shaded blocks indicatequantization matrices already encoded in the SPS and PPS and theunshaded blocks indicate quantization matrices yet to be encoded. Thus,in this example. quantization matrices 0-23 are encoded in the SPS andquantization matrices 0-14 are already encoded in the PPS. Quantizationmatrix 15, which is assumed to be a non-default matrix, is the matrixcurrently being compressed using matrix prediction.

Any of quantization matrices 0-23 in the SPS and quantization matrices0-14 in the PPS may be used as the reference quantization matrix forquantization matrix 15. A 1-bit flag is included in the PPS tocommunicate to the decoder whether the reference matrix is selected fromthe SPS or the PPS. As shown in this example, matrix 20 in the SPS ormatrix 12 in the PPS may be selected as the reference quantizationmatrix for matrix 15. Thus, the reference matrix indicator included inthe PPS may be 15−20=−5 coded with signed exp-Golomb code if matrix 20in the SPS is selected and may be 15−12=3 coded with unsigned exp-Golombcode if matrix 12 in the PPS is selected.

TABLE 13 Desc  1 pic_parameter_set_rbsp( ) {  2 pic_parameter_set_idue(v)  3 seq_parameter_set_id ue(v)  4 entropy_coding_mode_flag u(1)  5num_ref_idx_l0_default_active_minus1 ue(v)  6num_ref_idx_l1_default_active_minus1 ue(v)  7 pic_init_qp_minus26 /*relative to 26 */ se(v)  8 constrained_intra_pred_flag u(1)  9pic_scaling_matrix_present_flag u(1) 10 if(pic_scaling_matrix_present_flag ) 11 for( k = 0; k <=log2_diff_max_min_transform_block_size*6; k++ ) { 12  i = k +log2_min_transform_block_size_minus2*6 13 if ((i%6) == 0)qmatx_compressionID[i/6] u(3) 14 pic_scaling_list_present_flag[ i ] u(1)15 if( seq_scaling_list_present_flag[ i ] ) 16 if( i < 6 ) 17scaling_list( ScalingList4x4[ i ], 4, i, 0, qmatx_compressionID[i/6]) 18else(i < 12) 19 scaling_list( ScalingList8x8[ i − 6 ], 8, i, 0,qmatx_compressionID[i/6]) 20 else if (i < 18) 21 scaling_list(ScalingList16x16[ i − 12 ], 16, i, 0, qmatx_compressionID[i/6] ) 22 else23 scaling_list( ScalingList32x32[ i − 18 ], 32, i, 0,qmatx_compressionID[i/6] ) 24 } 25 rbsp_trailing_bits( ) 26 }

Referring again to FIG. 5, the order k of the exp-Golomb code used forcoding the 1D residual sequence and whether signed or unsignedexp-Golomb was used is also communicated to the decoder. For example, a2-bit flag may be used to signal the order of exp-Golomb code used, anda 1-bit flag may be used to signal whether signed or un-signedexp-Golomb code is used. Further, the decoder is informed as to whether1D sample prediction or matrix prediction is used. For example, a 1-bitflag may be used to signal this choice.

Tables 14 and 15 show examples of the quantization matrix compression ofFIG. 5 applied to an 8×8 quantization matrix. Both examples apply 135degree symmetry processing and 45 degree symmetry processing and useunsigned 1D prediction with unsigned kth order exp-Golomb coding of the1D residual sequence. The example of Table 15 also appliesdown-sampling.

TABLE 14 Original 8 × 8 quantization matrix 8 × 8 zigzag scanning matrix8, 11, 23, 26, 50, 53, 89, 92, 0, 1, 5, 6, 14, 15, 27, 28, 11, 20, 29,47, 56, 86, 95, 134, 2, 4, 7, 13, 16, 26, 29, 42, 23, 29, 44, 59, 83,98, 131, 137, 3, 8, 12, 17, 25, 30, 41, 43, 26, 47, 59, 80, 101, 128,140, 167, 9, 11, 18, 24, 31, 40, 44, 53, 50, 56, 83, 101, 125, 143, 164,170, 10, 19, 23, 32, 39, 45, 52, 54, 53, 86, 98, 128, 143, 161, 173,188, 20, 22, 33, 38, 46, 51, 55, 60, 89, 95, 131, 140, 164, 173, 185,191, 21, 34, 37, 47, 50, 56, 59, 61, 92, 134, 137, 167, 170, 188, 191,197, 35, 36, 48, 49, 57, 58, 62, 63, After 135 degree symmetryprocessing After 45 degree symmetry processing 8, 11, 23, 26, 50, 53,89, 92, 8, 11, 23, 26, 50, 53, 89, 92, 0, 20, 29, 47, 56, 86, 95, 134,0, 20, 29, 47, 56, 86, 95, 0, 0, 0, 44, 59, 83, 98, 131, 137, 0, 0, 44,59, 83, 98, 0, 0, 0, 0, 0, 80, 101, 128, 140, 167, 0, 0, 0, 80, 101, 0,0, 0, 0, 0, 0, 0, 125, 143, 164, 170, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,0, 0, 161, 173, 188, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 185, 191,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 197, 0, 0, 0, 0, 0, 0, 0,0, After zigzag scanning After 1-D prediction: residual sequence 8, 11,0, 0, 20, 23, 26, 29, 8, 3, 9, 3, 3, 3, 15, 3, 0, 0, 0, 0, 44, 47, 50,53, 3, 3, 3, 3, 21, 3, 3, 3, 56, 59, 0, 0, 0, 0, 0, 0, 3, 3, 3, 3, 80,83, 86, 89, 92, 95, 98, 101, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, Error ofreconstructed quantization matrix Reconstructed quantization matrix(Qsum = 205) (Qsum = 205) 0, 0, 0, 0, 0, 0, 0, 0, 8, 11, 23, 26, 50, 53,89, 92, 0, 0, 0, 0, 0, 0, 0, 18, 11, 20, 29, 47, 56, 86, 95, 116, 0, 0,0, 0, 0, 0, 12, −15, 23, 29, 44, 59, 83, 98, 119, 152, 0, 0, 0, 0, 0, 6,−9, 12, 26, 47, 59, 80, 101, 122, 149, 155, 0, 0, 0, 0, 0, −3, 6, −9,50, 56, 83, 101, 125, 146, 158, 179, 0, 0, 0, 6, −3, 0, −3, 6, 53, 86,98, 122, 146, 161, 176, 182, 0, 0, 12, −9, 6, −3, 0, −3, 89, 95, 119,149, 158, 176, 185, 194, 0, 18, −15, 12, −9, 6, −3, 0, 92, 116, 152,155, 179, 182, 194, 197,

TABLE 15 Original 8 × 8 quantization matrix Down-sampled 5 × 5quantization matrix 8, 11, 23, 26, 50, 53, 89, 92, 8, 11, 26, 53, 92,11, 20, 29, 47, 56, 86, 95, 134, 11, 20, 47, 86, 134, 23, 29, 44, 59,83, 98, 131, 137, 26, 47, 80, 128, 167, 26, 47, 59, 80, 101, 128, 140,167, 53, 86, 128, 161, 188, 50, 56, 83, 101, 125, 143, 164, 170, 92,134, 167, 188, 197, 53, 86, 98, 128, 143, 161, 173, 188, 89, 95, 131,140, 164, 173, 185, 191, 92, 134, 137, 167, 170, 188, 191, 197, After135 and 45 degree symmetry 5 × 5 zigzag scanning matrix processing (step2) plus step 3)) 0, 1, 5, 6, 14, 8, 11, 26, 53, 92, 2, 4, 7, 13, 15, 0,20, 47, 86, 0, 3, 8, 12, 16, 21, 0, 0, 80, 0, 0, 9, 11, 17, 20, 22, 0,0, 0, 0, 0, 10, 18, 19, 23, 24, 0, 0, 0, 0, 0, After zigzag scanningAfter 1-D prediction: residual sequence 8, 11, 0, 0, 20, 8, 3, 9, 6, 27,0, 27, 6, 6, 26, 53, 47, 0, 0, 0, 0, 80, 86, 92, 0, 0, 0, 0, 0, 0, 0, 0,0, 0, Reconstructed 5 × 5 quantization matrix Reconstructed 8 × 8quantization matrix after up-sampling (Qsum = 205) (Qsum = 205) 8, 11,26, 53, 92, 8, 11, 19, 26, 40, 53, 73, 92, 11, 20, 53, 86, 152, 11, 20,34, 47, 67, 86, 110, 134, 26, 53, 80, 152, 179, 19, 34, 49, 64, 86, 107,129, 151, 53, 86, 152, 185, 194, 26, 47, 64, 80, 104, 128, 148, 167, 92,152, 179, 194, 197, 40, 67, 86, 104, 125, 145, 162, 178, 53, 86, 107,128, 145, 161, 175, 188, 73, 110, 129, 148, 162, 175, 184, 193, 92, 134,151, 167, 178, 188, 193, 197,

Table 16 summarizes experimental results of applying the compressionmethod of FIG. 5 with various options selected to compress the examplequantization matrices of Tables 2-5. In Table 16, level 0 compressionincludes only zigzag scanning, signed 1D prediction, and signed kthorder exp-Golomb coding. Level 1 adds 135 degree symmetry processing inaddition to the processing of level 0. Level 2 adds 45 degree symmetryprocessing in addition to the processing of levels 0 and 1. Level 3compression includes both 135 and 45 degree symmetry processing and bothsigned 1D prediction and unsigned 1D prediction are enabled. Level 4compression includes all processing options except matrix prediction. Asshown in Table 16, the application of 135 degree symmetry (level 1)provides about 2× compression over level 0 and the compression ratiogoes up to roughly 4× when 135 degree symmetry, 45 degree symmetry, andboth signed and unsigned 1D prediction are enabled (level 3). Thecompression ratio goes over 7× if the down-sampling is also enabled(level 4).

TABLE 16 Quantization matrix block H.264 Level 0 Level 1 Level 2 Level 3Level 4 size # of bits # of bits # of bits # of bits # of bits # of bits4 × 4 140 118 80 62 48 38 8 × 8 344 298 178 114 82 55 16 × 16 702 704356 206 182 102 32 × 32 1578 1580 866 467 430 185 Total 2764 2700 1480849 742 380 Compression 1.02 1.87 3.26 3.73 7.27 ratio

If the quantization matrix is down-sampled for compression. up-samplingis needed for decompression of the matrix. Any suitable up-samplingtechnique may be used. Table 17 is example pseudo code for one possibleup-sampling technique. Note that this up-sampling technique isnormative.

TABLE 17 upSampleMatx(N, deciQmatx, Qmatx){ // horizontal bilinearmatrix up-sampling  for (i=0; i< N/2 +1 ; i++) { if (i > 1) row = i*2−1; else row = i;  for (j=0; j<N; j++) { if (j >1 & ((j&0x1) ==0) ) { Qmatx[row*N+j] =  (deciQmatx[i*(N/2+1)+j/2+1] + deciQmatx[i*(N/2+1)+j/2] + 1)>>1;  } else {   if (j < 2) Qmatx[row*N+j] = deciQmatx[i*(N/2+1)+j];  else  Qmatx[row*N+j] =deciQmatx[i*(N/2+1)+j/2+1]; }}} // vertical bilinear matrix up-sampling for (i=2; i< N ; i+=2)   for (j=0; j<N; j++) {  Qmatx[i*N+j] =(Qmatx[(i−1)*N+j] + Qmatx[(i+1)*N+j] +1 )>>1;}}

As was previously discussed, a reference quantization matrix may need tobe up-sampled either in a video encoder or a video decoder. FIGS. 8A-8Cshow an example of an N/2×N/2 to N×N up-sampling technique that may beused. In this example, FIG. 8A is the original matrix, FIG. 8B is thematrix after horizontal up-sampling, and FIG. 8C is the matrix aftervertical up-sampling. Further, horizontal up-sampling is performedbefore vertical up-sampling, and the following bilinear interpolationtechnique is used:

${\quad\left\{ \begin{matrix}{{e = \left( {{21*A} + {11*B} + 16} \right)}\operatorname{>>}5} \\{{f = \left( {{11*A} + {21*B} + 16} \right)}\operatorname{>>}5} \\{{g = \left( {B + C + 1} \right)}\operatorname{>>}1} \\{{h = \left( {C + D + 1} \right)}\operatorname{>>}1}\end{matrix} \right.}$where A, B, C, and D are sample values of a row in the originalquantization matrix and e, f, g, and h are interpolated samples afterthe horizontal matrix up-sampling.

As shown in FIG. 8B, in each row of the original quantization matrix(FIG. 8A), two samples, i.e., e and f, are interpolated between thefirst two samples, and a single sample, i.e., g and h, is interpolatedeach of the following pairs of samples using bilinear interpolation. Thesame process is applied to each row of the original matrix.

The vertical matrix up-sampling follows similar bilinear interpolationrules:

$\quad\left\{ \begin{matrix}{{i = \left( {{21*D} + {11*H} + 16} \right)}\operatorname{>>}5} \\{{j = \left( {{11*D} + {21*H} + 16} \right)}\operatorname{>>}5} \\{{k = \left( {H + L + 1} \right)}\operatorname{>>}1} \\{{l = \left( {L + O + 1} \right)}\operatorname{>>}1}\end{matrix} \right.$

where D, H, L, and O are sample values of a column in the quantizationmatrix after horizontal up-sampling and i, j, k, and I are interpolatedsamples after the vertical matrix up-sampling. As shown in FIG. 8C, ineach column of the so quantization matrix after horizontal up-sampling(FIG. 8B), two samples, i.e., i and j, are interpolated between thefirst two samples, and a single sample, i.e., k and I, is interpolatedbetween of the following pairs of samples using bilinear interpolation.The same process is applied to each matrix column.

Application of this up-sampling process one time up-samples an N×Nquantization matrix to a 2N×2N quantization matrix, e.g., from 8×8 to16×16. If an N×N quantization matrix needs to be up-sampled to 4N×4Ne.g., from 8×8 to 32×32, the above up-sampling process may be appliedtwice, e.g., for up-sampling an 8×8 matrix to a 32×32 matrix, the 8×8matrix is first up-sampled to 16×16 and then resulting 16×16 matrix isup-sampled to create the 32×32 matrix. In general, by applying the abovequantization matrix up-sampling process n times, an N×N matrix can beup-sampled to a (2″×N)×(2″×N) matrix. Vertical up-sampling may also beperformed before horizontal up-sampling.

FIG. 9A shows a flow diagram of a method for decompressing aquantization matrix in a video decoder. In general, the decompressionoperates to reverse the compression method performed by the videoencoder to reconstruct the quantization matrix. An embodiment of themethod may be performed, for example, by the quantization matrixdecompression component 401 of FIG. 4. The input to the method is acompressed quantization matrix, the quantization matrix compression ID,e.g., qmatx_compressionID of Tables 11 and 13, and the other overheadinformation previously described.

Initially, the overhead information for the compressed quantizationmatrix is decoded 900. The overhead information decoded may include a2-bit flag which signals the order of signed or un-signed exp-Golombcode used in compressing the quantization matrix, a 1-bit flag whichsignals whether signed or unsigned exp-Golomb code was used for residualcoding, the offset for the 45 degree symmetry processing (if used), anda 1-bit prediction type flag which signals whether 1D prediction ormatrix prediction was used. If the compressed quantization matrix is inthe SPS and matrix prediction was used for the compressed quantizationmatrix, unsigned exp-Golomb code is used to decode a reference matrixindicator identifying the reference quantization matrix used for thematrix prediction. If the compressed quantization matrix is in a PPS andmatrix prediction was used for the compressed quantization matrix,unsigned exp-Golomb code is used to decode a reference matrix indicatoridentifying the reference quantization matrix used for the matrixprediction, a 1-bit flag is decoded which indicates whether thereference matrix is from the SPS or the PPS, and a reference matrixindicator is decoded. This indicator is decoded using signed exp-Golombcode if the 1-bit flag indicates the matrix is from the SPS and isdecoded using unsigned exp-Golomb code if the 1-bit flag indicates thematrix is from the PPS.

After decoding of the overhead information. signed or unsigned kth orderexp-Golomb decoding is applied to the compressed quantization matrix toreconstruct the 1D residual sequence 904. The value of k and whethersigned or unsigned exp-Golomb coding is to be used is determined fromthe decoded overhead information.

After the 1D residual sequence is reconstructed, either inverse 1Dprediction 908 or inverse matrix prediction 910 is selected 906 based onthe decoded prediction type flag. Inverse matrix prediction 908 reversesthe signed or unsigned 1D prediction performed when compressing thequantization matrix to reconstruct the 1D sequence. Inverse matrixprediction 910 reverses the matrix prediction performed when compressingthe quantization matrix to reconstruct the 1D sequence. The appropriatereference matrix is selected 912 from the SPS quantization matrices 914or the PPS quantization matrices 916 based on the decoded overheadinformation.

The reconstructed 1D sequence from inverse 1D prediction 908 or inversematrix prediction 910 is then selected 918 for inverse zigzag scanning920. The inverse zigzag scanning 920 reverses the zigzag scanningperformed when compressing the quantization matrix to reconstruct thequantization matrix. 45 degree symmetry processing 922, 135 degreesymmetry processing 924, and/or up-sampling 926 may then be applied tothe reconstructed quantization matrix to generate the decoded 2Dquantization matrix. The value of the quantization matrix compression IDdetermines which, if any, of these are to be applied.

FIG. 9B shows a flow diagram of a method for decompressing aquantization matrix in a video decoder. This method assumes thatnon-default 4×4 and 8×8 quantization matrices are compressed by thevideo encoder and signaled in the compressed bit stream. The compressionof the 4×4 and 8×8 quantization matrices may be performed as previouslydescribed herein or may be performed using any other suitablecompression technique. An embodiment of the method may be performed, forexample, by the quantization matrix decompression component 401 of FIG.4.

Initially, the compressed bit stream is decoded 940. If a compressed 4×4quantization matrix is found in the compressed bit stream, thequantization matrix is decompressed 942 and output 952. Similarly, if acompressed 8×8 quantization matrix is found in the compressed bitstream, the quantization matrix is decompressed 944 and output 952.Factor 2 up-sampling 946 is then performed on the 8×8 reconstructedquantization matrix to generate a 16×16 reconstructed quantizationmatrix. Factor 2 up-sampling 950 is then performed on the 16×16reconstructed quantization matrix to generate a 32×32 reconstructedquantization matrix that is then output 952. Any suitable technique maybe used for the up-sampling. One suitable technique is previouslydescribed herein in reference to FIGS. 8A-8C. Note that the up-sampledmatrices are automatically generated when an 8×8 compressed quantizationmatrix is received, and that the up-sampled quantization matrices are ofthe same type, i.e., Y, Cr, Cb, intra or inter, as the 8×8 quantizationmatrix.

Embodiments of the methods, encoders, and decoders described herein maybe implemented for virtually any type of digital system (e.g., a desktop computer, a laptop computer, a handheld device such as a mobile(i.e., cellular) phone, a personal digital assistant, a digital camera.etc.). FIG. 10 is a block diagram of a digital system 1000 (e.g., amobile cellular telephone) that may be configured to use techniquesdescribed herein.

As shown in FIG. 10, the signal processing unit (SPU) 1002 includes adigital signal processing system (DSP) that includes embedded memory andsecurity features. The analog baseband unit 1004 receives a voice datastream from the handset microphone 1013a and sends a voice data streamto the handset mono speaker 1013b. The analog baseband unit 1004 alsoreceives a voice data stream from the microphone 1014a or 1032a andsends a voice data stream to the mono headset 1014b or wireless headset1032b. The analog baseband unit 1004 and the SPU 1002 may be separateICs. In many embodiments. the analog baseband unit 1004 does not embed aprogrammable processor core, but performs processing based onconfiguration of audio paths, filters, gains, etc being setup bysoftware running on the SPU 1002.

The display 1020 may display pictures and video sequences received froma local camera 1028, or from other sources such as the USB 1026 or thememory 1012. The SPU 1002 may also send a video sequence to the display1020 that is received from various sources such as the cellular networkvia the RF transceiver 1006 or the Bluetooth interface 1030. The SPU1002 may also send a video sequence to an external video display unitvia the encoder unit 1022 over a composite output terminal 1024. Theencoder unit 1022 may provide encoding according to PAL/SECAM/NTSC videostandards.

The SPU 1002 includes functionality to perform the computationaloperations required for video encoding and decoding. In one or moreembodiments, the SPU 1002 is configured to perform computationaloperations for applying one or more techniques for quantization matrixcompression and decompression during the encoding process as describedherein. Software instructions implementing all or part of the techniquesmay be stored in the memory 1012 and executed by the SPU 1002, forexample, as part of encoding video sequences captured by the localcamera 1028. The SPU 1002 is also configured to perform computationaloperations for applying one or more techniques for quantization matrixdecompression as described herein as part of decoding a received codedvideo sequence or decoding a coded video sequence stored in the memory1012. Software instructions implementing all or part of the techniquesmay be stored in the memory 1012 and executed by the SPU 1002.

Other Embodiments

While the invention has been described with respect to a limited numberof embodiments, those skilled in the art, having benefit of thisdisclosure, will appreciate that other embodiments can be devised whichdo not depart from the scope of the invention as disclosed herein. Forexample, in some embodiments, rather than compressing large quantizationmatrices, e.g., 16×16 and 32×32 matrices, the video encoder may select apreviously signaled smaller quantization matrix, e.g., a 4×4 or 8×8matrix, as a reference matrix for the larger matrix and signal in an SPSor PPS that the larger quantization matrix is to be reconstructed byup-sampling the smaller reference quantization matrix. The derivation ofthe larger matrices may be performed using any suitable technique, suchas up-sampling, interpolation, prediction, or any combination thereof.For example, the up-sampling technique of FIGS. 8A-8C may be used.

In another example, more or fewer quantization matrix sizes and/ordifferent quantization matrix sizes may be used.

In another example, rather than compressing quantization matrices forthe chroma components Cr and Cb. these quantization matrices may bederived from a Y component quantization matrix or vice versa.

Embodiments of the methods, encoders, and decoders described herein maybe implemented in hardware, software, firmware, or any combinationthereof. If completely or partially implemented in software. thesoftware may be executed in one or more processors, such as amicroprocessor, application specific integrated circuit (ASIC), fieldprogrammable gate array (FPGA), or digital signal processor (DSP). Thesoftware instructions may be initially stored in a computer-readablemedium and loaded and executed in the processor. In some cases, thesoftware instructions may also be sold in a computer program product,which includes the computer-readable medium and packaging materials forthe computer-readable medium. In some cases, the software instructionsmay be distributed via removable computer readable media, via atransmission path from computer readable media on another digitalsystem, etc. Examples of computer-readable media include non-writablestorage media such as read-only memory devices, writable storage mediasuch as disks, flash memory, memory, or a combination thereof.

It is therefore contemplated that the appended claims will cover anysuch modifications of the embodiments as fall within the true scope ofthe invention.

What is claimed is:
 1. A method of processing video, comprising:dividing a picture into a plurality of non-over-lapping blockscomprising residual pixel values; transforming the residual pixel valuesof the plurality of non-over-lapping blocks into transform coefficientsusing a transform operation; encoding a first flag into an encoded bitstream indicating a default quantization matrix or a non-defaultquantization matrix; determine a second index position where thenon-default quantization matrix at a first index position will use;quantizing the transform coefficients into quantized transformcoefficients using the non-default quantization matrix at the secondindex position and encoding a relationship between the first and secondindex position into the encoded video bit stream; and encoding thequantized transform coefficients into the encoded video bit stream. 2.The method of claim 1 further comprising scanning the non-default matrixin a one dimensional sequence for the non-default matrix and encodingthe one dimensional sequence for the non-default matrix into the encodedvideo bit stream.
 3. The method of claim 1 wherein the encoded video bitstream is part of a sequence parameter set (SPS) or a picture parameterset (PPS).
 4. The method of claim 2 wherein the encoded video bit streamis part of a sequence parameter set (SPS) or a picture parameter set(PPS).
 5. The method of claim 1 wherein the relationship between thefirst and second index position is a difference.
 6. The method of claim2 wherein the relationship between the first and second index positionis a difference.
 7. The method of claim 4 wherein the relationshipbetween the first and second index position is a difference.
 8. Themethod of claim 1 where the relationship between the first and secondindex position is encoded using a Golomb code.
 9. The method of claim 6where the difference is encoded using a Golomb code.
 10. The method ofclaim 1 further comprising capturing the picture using a charge coupleddevice (CCD) camera or a complementary metal oxide semiconductor (CMOS)camera.
 11. A method of decoding an encoded video bit stream,comprising: entropy decoding the encoded video bit stream resulting in aplurality of decoded video bits comprising a plurality of quantizedtransform coefficients; extracting a first flag from the decoded bitstream indicating a non-default quantization matrix; extracting a secondflag from the decoded bit stream indicating a relationship between afirst and second index position where the non-default quantizationmatrix will be implemented; performing an inverse quantization operationon the plurality of quantized transform coefficients using thenon-default quantization matrix at either the first or second indexposition resulting in a plurality of transform coefficients; andperforming an inverse transform operation on the plurality of transformcoefficients to form a plurality of reconstructed non-over-lappingblocks.
 12. The method of claim 11 further comprising recovering thenon-default quantization matrix in a first portion of the entropydecoded video bit stream.
 13. The method of claim 12 wherein the firstportion of the entropy decoded bit stream is for sequence parameter sets(SPS).
 14. The method of claim 12 wherein the first portion of theentropy decoded bit stream is for picture parameter sets (PPS).
 15. Themethod of claim 11 wherein the relationship is a difference.
 16. Themethod of claim 15 11wherein the non-default quantization matrix is an4×4 matrix, an 8×8 matrix or a 16×16 matrix.
 17. The method of claim 1511further comprising forming a frame using the plurality ofreconstructed non-over-lapping blocks and displaying the frame on adisplay.
 18. A system, comprising: a coding control component configuredto divide a picture into a plurality of non-overlapping blockscomprising residual pixel values; a transform component configured totransform the residual pixel values of the plurality of non-over-lappingblocks into transform coefficients using a transform operation; anentropy encoder component configured to encode a first flag into anencoded bit stream indicating a default quantization matrix or anon-default quantization matrix; a quantization component configured to:determine a second index position where the non-default quantizationmatrix at a first index position will use; and quantize the transformcoefficients into quantized transform coefficients using the non-defaultquantization matrix at the second index position; and the entropyencoder component further configured to: encode a relationship betweenthe first and second index position into the encoded video bit stream;and encode the quantized transform coefficients into the encoded videobit stream.
 19. The system of claim 18 wherein: the quantizationcomponent is further configured to scan the non-default matrix in a onedimensional sequence for the non-default matrix; and the entropy encodercomponent is further configured to encode the one dimensional sequencefor the non-default matrix into the encoded video bit stream.
 20. Thesystem of claim 18 wherein the encoded video bit stream is part of asequence parameter set (SPS) or a picture parameter set (PPS).
 21. Thesystem of claim 19 wherein the encoded video bit stream is part of asequence parameter set (SPS) or a picture parameter set (PPS).
 22. Thesystem of claim 18 wherein the relationship between the first and secondindex position is a difference.
 23. The system of claim 19 wherein therelationship between the first and second index position is adifference.
 24. The system of claim 21 wherein the relationship betweenthe first and second index position is a difference.
 25. The system ofclaim 18 where the relationship between the first and second indexposition is encoded using a Golomb code.
 26. The system of claim 23where the difference is encoded using a Golomb code.
 27. The system ofclaim 18 further comprising a charge coupled device (CCD) cameraconfigured to capture the picture.
 28. The system of claim 18 furthercomprising a complementary metal oxide semiconductor (CMOS) cameraconfigured to capture the picture.
 29. A system for decoding an encodedvideo bit stream, comprising: an entropy decoding component configuredto entropy decode the encoded video bit stream resulting in a pluralityof decoded video bits comprising a plurality of quantized transformcoefficients; and an inverse quantization component configured to:extract a first flag from the decoded bit stream indicating anon-default quantization matrix; extract a second flag from the decodedbit stream indicating a relationship between a first and second indexposition where the non-default quantization matrix will be implemented;perform an inverse quantization operation on the plurality of quantizedtransform coefficients using the non-default quantization matrix ateither the first or second index position resulting in a plurality oftransform coefficients; and perform an inverse transform operation onthe plurality of transform coefficients to form a plurality ofreconstructed non-over-lapping blocks.
 30. The system of claim 29wherein the entropy decoding component is further configured to recoverthe non-default quantization matrix in a first portion of the entropydecoded video bit stream.
 31. The system of claim 30 wherein the firstportion of the decoded bit stream is for sequence parameter sets (SPS).32. The system of claim 30 wherein the first portion of the entropydecoded bit stream is for picture parameter sets (PPS).
 33. The systemof claim 29 wherein the relationship is a difference.
 34. The system ofclaim 29 wherein the non-default quantization matrix is a 4×4 matrix, an8×8 matrix, or a 16×16 matrix.